Performance Improvement of DAG-Aware Task Scheduling Algorithms with Efficient Cache Management in Spark

Directed acyclic graph (DAG)-aware task scheduling algorithms have been studied extensively in recent years, and these achieved significant performance improvements data-parallel analytic platforms. However, current DAG-aware algorithms, among which HEFT GRAPHENE are notable, pay little attention to the cache management policy, plays a vital role in-memory systems such as Spark. Cache policies that designed for Spark exhibit poor task-scheduling leads misses degradation. In this study, we propose new policy known Long-Running Stage Set First (LSF), makes full use of dependencies optimize algorithms. LSF calculates caching prefetching priorities resilient distributed datasets according their unprocessed workloads significance parallel scheduling, key factors Moreover, present cache-aware algorithm based on reduce resource fragmentation computing. Experiments demonstrate that, compared with LRU MRD, same improve JCT by up 42% 30%, respectively. The proposed also exhibits about 12% reduction average job completion time LSF.